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Introduction 



One of the central issues in the national education debate is how to define, measure, and 
improve the quality of our schools. Among the key indicators of school quality are the extent, 
appropriateness, and accessibility of learning opportunities for students. Opportunities for students to 
learn important concepts and develop essential skills provide the foundation for them to become full 
and responsible members of society and productive participants in the nation’s workforce. A major 
determinant of opportunity to learn is the quality of the student-teacher interaction occurring daily 
within the classroom; the particular classroom instructional techniques and processes used by teachers 
directly and immediately influence student learning. Yet despite past work at the state, national, and 
international levels, our knowledge of what actually happens during that interaction is extremely 
limited; further refinements and greater sophistication are needed to generate, analyze, and interpret 
data on classroom instructional practices. 

The purpose of this paper is to describe the current efforts to collect data on classroom 
instructional processes and to recommend ways to advance those efforts. The paper begins with a 
review of current survey methods for gathering such data; it continues with an evaluation of their 
appropriateness for use in the Teacher Followup Survey (TFS); and it concludes with a 
recommendation for the 1994-95 TFS of specific types of survey items that would provide national 
estimates of important elements of classroom instructional processes. The paper includes a summary 
of issues in the definition and scope of instructional processes, the importance and usefulness of 
classroom descriptive data; then it briefly establishes the criteria by which existing approaches to data 
collection are evaluated. Those criteria are: relationship to student achievement, relevance to policy, 
appropriateness for the TFS sample, and item level of specificity. Within the parameters of four 
critical dimensions encompassing the instructional process and related elements-pedagogy, 
professional development, instructional materials and technologies, and topical coverage within 
courses— existing measurement approaches are presented, described, and analyzed. Finally, by 
employing the evaluation criteria to assess the appropriateness and applicability of these approaches 
for incorporation in the TFS, we recommend an approach suited to the particular requirements of that 
nationally representative sample. 

This paper builds on survey and other research reports from five sources: 

• National Center for Education Statistics (NCES) 

• Third International Mathematics and Science Study (TIMSS) and the Survey of 
Mathematics and Science Opportunities (SMSO) 
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• Reform Up Close (RUC) 

• RAND/University of California, Los Angeles (UCLA): The School Reform 
Assessment Project (SRA), Validating National Curriculum Indicators (VNCI), and 
the California Learning Assessment System (CLAS) 

• Council of Chief State School Officers: State Collaborative on Assessment and 
Student Standards (SCASS) 

Survey instruments are listed in Appendix A. 



Scope, Importance, and Usefulness of Instructional Process Data 



The observations of two researchers with prominent work in this area represent recent and 
current research efforts to define the boundaries of classroom instructional processes. Andrew Porter 
and colleagues working on the RUC study characterize student opportunity to learn as a function of 
"the enacted curriculum as experienced by the student" (Porter, May 1993, p. 7). His definition of 
the enacted curriculum includes not only the "content of instruction" (the particular skills, concepts, 
and applications taught), but also the "pedagogical quality" of instruction, and the "resources available 
to students and teachers as instruction takes place." This broad definition encompasses, but is not 
limited to, the following elements: the specific topics taught; influences on the teacher in selecting 
topics and concepts to be taught; the emphasis applied to the topic; the time allocated to it; the 
teacher’s knowledge of the topic; the textbook’s treatment of it; the student’s prior learning before 
instruction begins; the level of anticipated student mastery; the test items developed for assessing 
student mastery; and any interdisciplinary treatment of the topic. Not included in this definition are 
other elements of the schooling process such as school leadership, school goals, parent and 
community support, district and state support, or quality of school life. 

William Schmidt and colleagues working on TIMSS and its predecessor, the SMSO project, 
have also developed related definitions that distinguish among types of curricula. For TIMSS/SMSO, 
in the intended curriculum, content is defined at the system level; in the implemented curriculum, 
content is interpreted by teachers and made available to students; while the attained curriculum is 
content students have learned in the course of their studies (TIMSS, August 1991). The SMSO 
operational definition distinguished between the learning opportunities afforded students and the 
classroom processes and practices making that learning possible (Burstein, Guiton, Bayley, Isaacson, 
September 1991). Within that definition, classroom processes and practices include instructional goals 
and objectives, the organization of instruction, the instructional activities of both students and 



teachers, teacher pedagogical beliefs, and teacher knowledge. Learning opportunities, on the other 
hand, include the specific content covered, the teacher’s emphasis on the topic, and the allocated time. 
Coverage indicates what material was covered, for whom, and in what grade level; emphasis refers to 
the importance teachers place on particular instructional goals; and time allocations indicate the 
number of classroom hours or class periods allocated to specific content subtopics. 

Given the dynamic nature of the teacher-student-topic interaction, the SMSO researchers also 
observed that the process itself and the resulting opportunity to learn are influenced not only by 
elements of each dimension above, but by characteristics of the teacher, student, and the setting. 

These important influences include the background characteristics of the teacher (subject-matter 
knowledge, for example) and the student (prior learning experiences, for example), plus the physical 
features of the school such as appropriate facilities (Schmidt, no date). 

Although the outside boundaries of "classroom instructional practices" may be circumscribed, 
no single accepted definition of the phrase prevails. However, there is agreement on the importance 
and usefulness of a central core of school and educational process indicators. As high-stakes testing 
becomes more prominent and greater attention is placed on student and school outcomes, the desire to 
understand variation among students and groups of students (by race or ethnic group, income level, or 
geographic region) will focus attention on variation in classroom instructional practices and on the 
resulting opportunities of students to learn the material being evaluated. In California, for example, 
the California Assessment Policy Committee (cited in Bennett, 1994) suggests: "Scores for individual 
students should be accompanied by indicators of the degree to which they have had the opportunity to 
learn, i.e., to be exposed to the core curriculum." Systematic measures of classroom instructional 
processes are one method of evaluating this effectively. 

Porter and Schmidt do agree that it is important to assess two critical elements of classroom 
instructional practices— the content presented and the pedagogy employed— that together shape students’ 
classroom activities. Furthermore, they stress that the quality of any measure of those elements must 
be assessed for its "utility for predicting student learning. " Using that criterion. Porter suggests that 
content and pedagogy are the two best predictors of student achievement. Looking at the process 
from the students’ perspective, Schmidt agrees. He observes that the two most important measures of 
instructional opportunities are students’ exposure to particular content elements of the curriculum and 
students’ use of classtime. 

Porter further suggests that classroom instructional practices can be important indicators of 
school process; they can offer elements either to compose a vision of what ought to be or to establish 
a benchmark to chart progress on a wide scale. They provide descriptive information about what the 
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educational system is offering to students; they inform understanding of the match between types of 
students and types of education; and they are a means with which to explain the system’s success— or 
lack of success— at educating its children. 

David Hombeck, then co-director of the National Alliance for Restructuring Education, 
described four potential uses of such information about instruction (January 1994): 

• Identifying schools needing assistance 

• Relating school and district incentives appropriately to student gains in 
learning 

• Helping to choose sound curriculum and instructional materials 

• Improving professional development at the school and district level 

Finally, information on classroom instructional practices and students’ opportunities to learn 
resulting from those practices creates a common base of knowledge for dialogue about educational 
equity, the variation in student achievement attributable to differences in school and classroom 
educational opportunities, and the educational conditions associated with high performance. 
Understanding these factors is essential for creating equal opportunity to learn for all students. Given 
the importance of classroom process information, establishing a federal role in these early stages of 
exploration seems appropriate. According to Emerson J. Elliott, Commissioner of the National 
Center for Education Statistics, this role might include advancing the debate; furthering our 
understanding of what is worth knowing; and supporting the gathering of statistics— allowing one to 
know about the distribution of opportunities and perhaps to relate those opportunities to student 
achievement (June, 1994). 



Evaluation Criteria 

In this section we frame our analysis by describing the relevant criteria on which the existing 
measurement approaches will be evaluated. Those criteria are: 



relationship to student achievement 
relevance to policy 
appropriateness for the TFS sample 



level of item specificity 



• 

For any measure to be meaningful, useful, and worth knowing, it must have a hypothesized 
or established relationship to student achievement. Without such an association the data may well be 
irrelevant. Second, although a measure’s relationship to student achievement is important, if 
educational change is to come at a systemic level, its relevance to decision making and policy setting 
is crucial. Data from large scale surveys with carefully framed items can provide an accurate basis 
upon which intelligent policy can be established. To compile data of that quality, selected items must 
be appropriate for the TFS sample, a broadly representative survey of public and private teachers, 
including teachers who recently changed schools and those who have left the profession altogether. 
Survey language and content must be applicable to teachers of all grade levels, from prekindergarten 
to postsecondary. Finally, to obtain useful information from such a diverse sample, survey designers 
must carefully target each item’s level of specificity because items that probe deeply may be 
inappropriately content- or grade-level specific for the TFS sample, while more general questions may 
not reveal information of interest. 

These four criteria form the basis on which the existing measurement approaches to 
understanding classroom instructional processes will be evaluated. 



Description and Analysis of Existing Approaches 



Four dimensions within the broad realm of classroom instruction are critical: (1) pedagogy, 
(2) professional development, (3) instructional materials and technology, and (4) topical coverage 
within courses. These areas were identified as having potential value for professional and public 
awareness of classroom activities and the state of students’ opportunity to learn. This section presents 
a description and analysis of the current methodology for gathering data on these four dimensions of 
the educational process. 



Pf>Hayo<ry 

The greatest variety of questions in the surveys examined is in the area of classroom 
pedagogy. Many aspects of pedagogy are included because the term covers multiple overlapping 
processes. For this paper, pedagogy items have been classified into five areas: teacher actions; 
student actions; intended learner outcomes; organization (of the lesson or activity); and other 
instructional strategies, including homework and student assessment. 
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Teacher actions. Questionnaire items about the teacher’s actions in the classroom generally 
either probe a specific type of action and its motivations and intents, or cover a range of related 
actions and their frequencies of occurrence. Examples of questions about a specific action are items 
about teachers’ motivations and their techniques in questioning students. In two separate items using 
a frequency response option (never/rarely /sometimes/often), the TIMSS questionnaire seeks an 
understanding of the specific purpose for which teachers ask students questions and how teachers 
respond to an incorrect answer. In the first, teachers are asked to indicate the frequency with which 
they use questioning techniques for the following purposes: to develop a procedure in mathematics; 
develop a mathematical concept; determine students’ understanding of the lesson concept; and explore 
possible misunderstandings. In the second, teachers are asked the frequency with which they correct 
a student’s error in front of the class; ask the student another question to assist in getting the correct 
response; call on another student likely to give the correct response; call on other students, then 
discuss what is correct; or provide the correct answer. 

Questions seeking information about a range of related teacher actions are included in every 
survey instrument reviewed. The RAND/UCLA and SCASS questionnaires include a typical range of 
actions. In the first instance, teachers are queried about their use of eight modes of instruction, such 
as lecture to entire class; demonstrate an exercise at the board; use manipulatives or audio-visual 
materials to demonstrate a concept; demonstrate an experiment. In this particular questionnaire the 
directions ask teachers to check all that apply, but response options identifying frequency of use 
would certainly be possible and desirable. Similarly, the SCASS questionnaire includes nine teacher 
activities, each either directly or. indirectly related to instruction, such as providing instruction to the 
class as a whole; providing instruction to small groups of students; administering tests or quizzes; 
performing administrative tasks (e.g., taking attendance); doing other school activities not related to 
the subject. Responses are phrased in percentages of classtime during a typical week with six options 
ranging from none to 75 % + . 

Student actions. On the other side of the teacher-student interaction are questions about what 
students actually do during the process of learning. Nearly every questionnaire includes items seeking 
information about the frequency of a range of student actions in the classroom. Response options are 
dependent on the period of time covered by the questionnaire and vary between the amount of 
classtime in a week ("none/30 minutes/1 hour/2 hours/3 + hours") to the frequency over a year 
("almost every day/1-2 week/1-2 month/1-2 semester/1-2 year/never"). 

Interestingly, similar questions in two RUC questionnaires included the same six student 
activities (such as "listen or take notes/discuss/complete written exercises/write report or paper/lab or 
fieldwork/present or demonstrate"), but with different response options. One response option was the 
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level of emphasis (on a scale between 0 to 3) assigned by the teacher to each specific student activity; 
the other was an indication of the amount of classtime allocated to each. The number of student 
actions examined in any one list varied also, ranging from 6 to 20 in the questionnaires reviewed. 
Longer, and more detailed, lists of student actions tended to be on subject-matter-specific 
questionnaires. For example, the SRA math questionnaire has 14 items, some of which could be 
generic ("use hands-on materials/use a workbook or ditto sheet"), but most of which are math-specific 
("work mathematics problems at the board/use calculators in class"). The SCASS science instrument 
lists 20 student activities, some generic ("write answers to essay questions/take quizzes or tests with 
multiple-choice, true-false, matching, or short answer questions"), some with a specific application to 
science ("design their own experiment to test an hypothesis/use photographs or drawings to document 
science activities"), but nearly all amenable to other content areas ("do science projects or 
presentations alone/keep a collection of their science work"). 

Although in most questionnaires there is a clear and helpftil distinction between the actions of 
the teacher and those of the students, in some questions actions of the teacher and students were 
combined under a label such as "instructional strategies" describing both teacher actions ("lecture") 
and student actions ("work with manipulatives”). A clear example is the VNCI instrument with 26 
different events covering the entire range from a strictly teacher action ("administer a quiz"), to an 
organizational strategy ("lecture"), to an organizational strategy combined with student actions 
("student-led whole group discussions"), to student actions with intended learner outcomes ("have 
small groups work on problems to find a joint solution"). According to the instrument designers, the 
intent behind the organizing strategy was to group together a diverse collection of classroom activities 
and strategies, later to be separated and analyzed as "reform" ("students work on problems for which 
there is no obvious solution") or "non-reform" ("students work on next day’s homework in class") (L. 
McDonnell, personal communication. May 16, 1994). 

Intended learner outcomes. Another smaller, yet distinct group of questions investigates 
intended learner outcomes. Learner outcomes look less at observable actions of the teacher and 
student and more at the instructional intention or purpose behind the activity. Examples of such a 
questioning strategy come from the CLAS questionnaire in which teachers indicate the frequency with 
which they structure their classroom such that students are asked to do 12 strategies such as 
"generalize from patterns or examples; generate original examples; organize, summarize, and display 
data." Response options are "more than 5 class periods/1-5 class periods/not at all." 

Organization. Besides questions on student actions, teacher actions, and learner outcomes, 
instruments also probe the structure and order of a lesson or activity. Every questionnaire includes 
items that distinguish among six common methods of structuring teaching. The RUC survey, for 
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example, includes these options: lecture; demonstration; recitation/drill; whole class discussion; 
students working in pairs/teams/small groups; students working independently. With minor 
variations, this list is included in every questionnaire, accompanied by response options using 
frequency ("never/1-2 semester/1-2 month/1-2 week/almost every day"), proportion of the lesson 
(with open-ended responses), or level of emphasis assigned (on a scale of 0 to 3). 

The TIMSS survey takes a different approach to investigating instructional organization, 
asking teachers to indicate specific elements included in the lesson, their duration, and the order in 
which each occurred. A list of ten elements is included, such as "review of previous lesson; review 
of previous lesson’s homework; introduction of a topic; development of a topic; oral recitation; paper 
and pencil exercises; assignment of student homework; students work on homework in class; small 
group activities; laboratory or data collection activity." 

Other specific instructional strategies. Three other elements of classroom pedagogy are 
relevant here: homework, student assessment, and teacher’s influence. Each is described below. 

Homework. Four aspects of homework assignments are queried across most instruments: the 
frequency with which the teacher assigns homework, its type and length, and the uses to which the 
teacher puts the completed assignments. The TIMSS instrument separates questions about homework 
frequency from questions about other aspects of homework, simply asking the frequency with which 
teachers assigned homework during a recent typical week. Response options include "never/less than 
once a week/once or twice a week/3 or 4 times a week/almost every day. " Typically however, 
questions economically combine a request for homework frequency data with queries on the type, 
length, and uses of the homework. For example, the VNCI questionnaire requests information on 
both the type of homework and the frequency with which each type is assigned. The 13 homework 
types included range from reading ("the text or supplementary materials"), to writing (in a journal or 
completing exercises from a text), to gathering data or preparing reports, to applying concepts or 
extending results. Response options key teachers to reflect on a certain instructional period, usually 
the most recent complete week, but sometimes the past semester: "almost every day/once or twice a 
week/once or twice a month/once or twice a semester/never." Other response options range from 
open-ended (TIMSS) to 30-minute blocks covering 0 to 3 hours (RUC, SRA, VNCI) and were 
phrased generically, with no specific type of homework indicated. Interestingly, items in these 
instruments questioning the length of time teachers expect students to spend completing homework 
assignments include neither frequency responses nor types of homework. This approach seems to 
assume that every homework assigned is of the same length and that length is the same for every type 
of homework. What’s more, it has the perhaps unintended effect of assuming all homework 
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assignments, even for those with differing purposes, require the same amount of student time to 
complete. 

Items in nearly all surveys probe teachers about the uses to which teacher and students put the 
completed homework assignments. This includes not only whether assignments are graded and 
whether that grade is recorded, but also whether completed assignments are used instructionally. For 
example, besides five possible scenarios for correcting homework, the TIMSS questionnaire also asks 
whether teachers "use it as a basis for class discussion" or "as a basis for student assessment. " 
Responses are indicated in frequency options ("never/rarely/sometimes/often"). 

Student assessment. Questions on student assessment generally query either the type of 
assessment used or the uses to which teachers put assessment results. For both purposes, response 
options are stated in terms of frequencies or percentages of the total. In a series of three items, the 
VNCI survey makes detailed distinctions among facets of assessments. The first item differentiates 
among four student response options (multiple choice, short-answer, essay, open-ended, and other); 
the second item probes what assessments require students to do (recognize or recall definitions, 
explain reasoning, apply concepts to unfamiliar situations, etc.); the third item highlights exercise 
formats (minor variations of known problems, problems with more than one answer, more than one 
approach, more than one step necessary for solution). All items seek responses from teachers in 
terms of percentages of the total number of items on quizzes and tests. 

Information about the uses of assessments is gathered most comprehensively in the TIMSS 
questionnaire. It includes six possible uses to which teachers put assessment information. These 
include student grades, student feedback, parent feedback, diagnosing student learning problems, 
assigning students to programs or tracks, and planning for future lessons. Teacher response options 
are on a four-point scale from never to usually. 

One of the uses of assessment information is to determine student grades, and the RUC and 
TIMSS questionnaires probe this feature more deeply by seeking information about the importance of 
specific assessment criteria in determining how teachers establish individual student grades. Criteria 
included in the RUC instrument are absolute and relative levels of achievement, improvement, effort, 
class participation, homework completion, and attendance; the response options are "not 
important/somewhat important/very important/extremely important." The TIMSS instrument takes a 
different slant, including standardized tests, open-ended tests, objective tests made by the teacher, 
homework performance, performance on projects and labs, class participation, and teacher 
observation. Response options indicating levels of importance are "none/little/some/a lot." 
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Teacher’s influence on pedagogy. Surveys also contain a set of more general questions 
probing teachers’ opinions and approaches to conceptualizing the pedagogical process and classroom 
decision-making. For example, the TIMSS questionnaire probes teachers’ perception of their 
influence in structuring the daily lesson; in selecting the pedagogic strategy used; in the amount of 
homework assigned; and in how students’ grades are determined. 



Professional Development 

Information about teachers’ professional development activities is covered more extensively by 
NCES questionnaires than by the other surveys reviewed here. NAEP, NELS, and SASS-4A each 
include professional development questions scrutinizing the type and sponsorship of the activities, the 
specific topics covered, the amount of system support for the training, and an evaluation of the 
training’s utility. In addition, SASS-4A includes extensive questions on teacher training and 
preparation; however, those items are not detailed here. Because their purpose is to understand the 
preservice preparation and training of the teaching force, they shed no light on activities designed to 
maintain or upgrade teachers’ professional skills following certification or inception of teaching. 

Survey questions about the type and sponsorship of professional development activities 
generally include not only workshops, inservice programs, pertinent college-level courses, committee 
activities, and other growth activities, but also determine whether such activities are sponsored by 
school district, school, university, or professional associations. Although total time devoted to such 
activities is obviously of interest and queried on many instruments, most questions about type and 
sponsorship of professional activities are simple "yes/no" participation questions; the number of hours 
spent on each particular type of activity is questioned on only one survey— the more detailed and 
weekly RUC questionnaire. Several questionnaires, however, do probe the number of hours devoted 
to more specific development activities, such as lesson planning and informal meetings with other 
teachers. 

The second major area of interest is an understanding of the specific topics studied during 
professional development. The 1993-94 SASS-4A includes questions about educational technology, 
subject matter teaching methods, cooperative learning, student assessment, and in-depth subject matter 
study. These questions are also unique in that response options are constructed in number of hours (8 
hours or less/9-32 hours/more than 32 hours). 

The final area of interest is the amount of support for professional development activities 
provided teachers. For example, the NELS questionnaires for the base year and for the first and 



second follow-ups each seek (yes/no) information on whether teachers have received released time, 
travel money or per diem, stipends, or professional growth credits in support of their professio nal 
development. 



Instructional Materials and Technologies 

Questionnaire items probe the use of instructional materials and technology in four areas: 
texts and other printed materials; calculators and computers; science-related materials; and library and 
audio-visual materials. The intent of many questions is to determine the level of teacher satisfaction 
with the selection, amount, and use of these instructional materials. 

Texts and other printed materials . Research interest in textbooks generally concerns the type 
of reading materials used, the book selection process, the percent of the textbook covered during a 
course, and the frequency of its use. Instruments that collect information on the different types of 
texts and other reading materials used by teachers often combine those questions with questions on 
frequency of use, simply adding a response option of "never" to an array of frequency responses. 

The NELS:88 base year questionnaire and both TIMSS math and science questionnaires included an 
open-response item querying primary text use in which respondents provide specific details about 
title, author, publisher, date, and edition. Subsequent NELS questions have been more general, 
seeking only to distinguish teachers’ use of basal, tradebooks, both, or other materials as students’ 
primary reading materials. Three instruments ask whether there is participation in the textbook- 
selection process. More general information on teachers’ influence over textbook selection, what 
supplies are chosen, and the amount of money allocated for them are included in both TIMSS 
questionnaires. Questions on the NELS: 88 base-year questionnaire and more recently on the RUC 
Math/Science teacher questionnaire are more specific and nearly identical. Response options for the 
question, "Who helped choose text?" are "teacher/principal/group of teachers/department head/district 
committee/state committee." The same two instruments query teachers on the percent of the textbook 
covered during their course. Response options for both instruments are expressed in percentages, 
with NELS instrument options in ten percentage categories (such as 50-59%) while the RUC options 
are generally 25 percent categories (50-75%) with smaller high end options (75-90%/>90%). Both 
the NELS: 88 base year questionnaire and the First Followup use the same question on frequency of 
teacher’s use of texts and other reading materials with nearly identical response options of "primary 
resource, used frequently /secondary resource, used occasionally /rarely used/not used." The single 
difference in response options between the two administrations of the questionnaire is the change in 
the First Followup’s descriptive phrase of response option two, recognizing that a secondary resource 
can be used "frequently" (not just "occasionally" as in the 1988 original). The NAEP questionnaires 
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are the only other instruments gathering information about frequency of text use. In separate 
questions this instrument collects information on the use of reading texts and materials from other 
content areas. The 1993-94 Grade 4 NAEP included a new and similar item on the classroom use of 
children’s magazines. 

Calculators and computers. Every survey except SASS and TFS seeks information about 
computer and calculator availability and use. Calculator questions range from determining their 
availability, prevalence, and ownership, to their frequency of use. Several questionnaires go further 
to determine the purpose of the calculator use. RUC lists 11 possible uses; SRA lists five uses; and 
TIMSS lists the same five possible uses in both math and science. Questions about computers are 
more limited, with most identifying computer availability (RUC, NAEP, NELS), and two determining 
computer use (SRA, RUC). SRA collects computer use information with three response options and 
RUC offers 1 1 possible options ("teacher demonstrating computer use/writing programs/leaming math 
content/laboratory tool/drill and practice/using simulations/problem solving/using computer 
graphics/games/testing and evaluation/homework"). One survey gathers information on the use of 
computer software for reading instruction (NAEP), and one collects information on the effect of 
computer shortages (TIMSS). 

Science-related materials. Most science-related materials and technology questions are 
included only in the NELS family of questionnaires. Indeed, only the NELS Baseline and Second 
Followup contain questions about science laboratory access or availability, and the amount and 
condition of science equipment. The Second Followup also asks a similar question on consumable 
science supplies. Surprisingly, the only instances of questions on science equipment in the TIMSS 
science survey consist of the teacher’s opinion on factors that limit classroom options, with two of the 
18 possible factors being a shortage of equipment for students and a shortage of equipment for the 
teacher’s use in demonstrations. SASS and TFS questions are necessarily general in their questions 
on materials and do not distinguish among topic areas; they group together materials and equipment 
queries for all content areas. 

Library and audio-visual materials. SASS collects wide-ranging information on library 
media center staffing, funding, technology, facilities, and scheduling through surveys of public, 
private, and Indian school libraries. The instruments gather information on the training, activities, 
compensation, and working conditions of the librarians or media specialists in those same schools. 
Additionally, the NAEP family of questionnaires seeks information about library access and frequency 
of use. 
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Topical Coverage within Courses 



One certainty from prior research in classroom instructional processes is that course titles by 
themselves provide little help in determining the content of the course. The content within courses 
having the same title varies in different districts, different schools, or even within the same school 
taught by different people. Course content also changes and differs depending on the extent of review 
and coverage of new material (L. Burstein, personal communication, May 16, 1994). Thus, 
questionnaire items about content coverage generally seek information in three areas: the specific 
topics, themes, or skills covered or expected to be covered by a teacher throughout a particular 
course; the topic’s emphasis within the course; and the amount of classtime allocated to that pursuit. 

Specific topics covered. Some questionnaire items investigate coverage alone, asking whether 
specific topics are reviewed or taught for the first time. Most subject matter-specific questions 
currently developed are contained in math and science surveys (TIMSS, SCASS, CLAS, VNCI), 
although there are questions in other surveys directed toward social studies, U.S. history, and 
English/language arts areas (SRA, NELS, NAEP). These questions usually use response options such 
as "taught as new content/reviewed only/assumed as prerequisite knowledge/not taught and not 
assessed as student knowledge" (SRA). 

Some questionnaires also gather more specific information about literature instruction in the 
classroom, asking for frequency of use of specific types of literature (such as fiction, poetry, 
mythology, etc.) in student assignments, with response options such as "majority of the time/some of 
the time/rarely /never" (NELS, NAEP). The NELS questionnaire also asked the number of books 
eighth-grade teachers require their students to read in a school year, with response options from 0 to 
5+. 



Two questionnaires also probe teachers’ influences in determining specific course content 
(SASS, RUC). For example, in Part II of a course-specific math teacher questionnaire, RUC asks 
teachers to indicate the type and level of influences on topic selection. Possible types of influence 
stem from five sources (guides, books, tests, administrators, or personal decisions) and levels of 
influence include "major/moderate/minor/none." 

NAEP and SASS distinguish among topics, themes, and skills, using a level of specificity 
more possible in some disciplines than in others. In social studies, for example, topics may be "the 
period before 1815" or "the period between 1815 and 1865"; themes could be the "development of 
American democracy and politics" or "the changing role of the United States in the world"; and skills 



are those such as "using maps and globes" or "finding places using latitude and longitude." 

However, themes and topics are usually combined and distinct from skills. 

The distinctions among topics, themes, and skills are also associated with different response 
options. In general, questions about topics have units of time as response options ("6+weeks/3-5 
weeks/1-2 weeks/not covered"), questions about themes have emphasis response options 
("major/moderate/ minor/none"), and questions about skills have frequency response options ("almost 
every day/1-2 week/1-2 month/never or hardly ever"). 

Indepth questions on content coverage require a longer questionnaire and entail a greater 
response burden. Employing 40 questions, the TIMSS/SMSO math and science questionnaires delve 
into more detail on topic coverage and other areas of inquiry than do most other questionnaires 
reviewed. For example, in the content-coverage questions directed at math teachers of 13-year-old 
students, teachers are presented with 22 math "topic areas," each defined with a list of from three to 
seven subtopics. Response options are keyed to the number of completed class hours ("1-5/6- 
10/1 1 + ") already spent teaching each topic and to the options indicating whether the topics will be 
covered in the future. In a subsequent, more detailed section, an additional 18 questions are 
presented on selected subtopics within some topic areas. 

Emphasis within the course. Some surveys request information on the level of emphasis 
classroom teachers give specific topics or objectives in their instruction. For each topic, teachers are 
asked to rate its emphasis in the course in degrees, such as "major/minor/review/not at all" (NELS) 
or "major/moderate/minor/none" (VNCI). Because there is a fine distinction between determining 
whether a topic is covered at all during a course or school year and assessing the emphasis a topic 
receives during teaching time, some surveys combine questions on whether it was covered at all with 
questions on the amount of time spent teaching it. For this type of question, response options are 
stated in units of time ("6+ weeks/3-5 weeks/ 1-2 weeks/not covered" (NAEP)), degrees of emphasis 
("major/minor/review/not at all" (NELS) or "major/moderate/minor/none" (NELS, VNCI)), or 
frequency of instruction ("almost every day/1-2 week/1-2 month/never or hardly ever" (NAEP, 
VNCI)). Researchers evaluating different response options suggest that respondents disagree on the 
meanings and distinctions attached to undefined degrees of emphasis such as "major-moderate-minor" 
and prefer response options presented in units of time or frequency of instruction (L. Burstein, 
personal communication, May 19, 1994). 

Furthermore, in this as in other questions, response options reflect the timing of the 
questionnaire’s administration and the period of time on which the teacher is asked to reflect, and the 
meaning attached to the word emphasis is also likely to be dependent on that timeframe. For 
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example, options in the CLAS questionnaire, administered at the end of the school year, range from 
"0-5/6-10/11 -20/more than 20 class periods." On the daily log form for the RUC project, math 
topics are first coded for specific areas of content from a possible array of 100 topics in a two-level 
content taxonomy. Then teachers are asked to evaluate the content emphasis of the day’s lesson, not 
that of the course or year’s topics. These choices of emphasis range from the "only content 
emphasized in the period (more than 50% of lesson)," to "one of 2-4 types of content emphasized in 
the period,” to "important content, but not emphasized in this lesson (less than 20%)." Thus the 
meaning of the word emphasis depends on the associated time period. 

Gasstime allocation. Questions on the allocation of classtime ask for responses in minutes 
per day or hours per week. In elementary grades, questions usually concern a general subject area; in 
middle and upper grades, questions are centered on specific topics within a course. Some 
questionnaires (VNCI, TIMSS) combine classtime allocation questions with topic treatment. Such 
coverage questions allow a researcher to relate the number of class periods allocated to a topic and to 
assess whether it is introduced as new material or whether it is only reviewed, for example. The 
accuracy and ultimate value of this information depend on the teacher’s ability to recall with precision 
the amount of time allocated to any given topic (CCSSO, February 25, 1992). Therefore, the utility 
of questions on specific amounts of time devoted to the teaching of a particular topic is heavily 
dependent on the length of time over which teachers are asked to recall. The function of such 
questions is limited, therefore, because respondents can recall events within the last semester or year, 
for example, less accurately than events within the last week or month. 

The distinction between topic emphasis and allocation of time may provoke differences in 
interpretations among respondents because a topic’s importance or emphasis in the classroom may be 
reflected in multiple ways. For example, some teachers may emphasize a particular topic by devoting 
a number of consecutive weeks to its teaching; others may emphasize a topic or skill by integrating it 
into many different units, returning repeatedly to it throughout the year, yet in the process collectively 
allocating a minimum of time to the topic. In examining the importance of topics in the curriculum, 
one must carefully interpret the distinction between topic emphasis and allocation of time. 

Two efforts (TIMSS/SMSO and CLAS) seek to relate topic coverage responses on teacher 
questionnaires with item results on student assessments. For example, content coverage questions in 
the SMSO teacher questionnaire can be directly related to the specific student concepts tested on 
TIMSS. Providing two examples of student assessment items, SMSO asks teachers whether they have 
done anything in their teaching to prepare students to respond correctly to those items. Teachers also 
indicate whether that topic is emphasized in class and whether students are likely to encounter the 
topic outside of class. 
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Summary Evaluation of Appropriateness and Applicability for TFS 



This section evaluates the critical dimensions of the instructional process-pedagogy, 
professional development, instructional materials and technology, and topical coverage within courses- 
-using the criteria described earlier-relationship to student achievement, relevance to policy, 
appropriateness for the TFS sample, and level of specificity. Recommendations are made in the next 
section. 



Relationship to Student Achievement 



All four dimensions under consideration have an established or expected relationship to 
student achievement and could provide interesting information about variation in achievement. Of the 
four, pedagogy and topical coverage within courses are more related to the teacher/student interaction 
and may therefore have a stronger relationship with student achievement. The interesting variation in 
professional development may be less in the number of hours and type of development experienced as 
revealed by data currently collected, and more in the motivation and receptivity of the teacher 
attending, and the specific topics, quality, and applicability of the experience, all data currently not 
collected. Similarly, the value and relevance of instructional materials to the educational process may 
vary depending on the subject matter involved. For example, the effect and relevance of the 
availability and use of textbooks may be more applicable to the teaching of reading than to the 
teaching of social studies, for example. The importance of access to and facility with a calculator 
may be more important to the process of learning algebra and advanced mathematics than to learning 
geometry. Finally, whether a teacher uses a text or other teacher-gathered materials such as primary 
documents, reference materials, journals, and periodicals to teach social studies may be less important 
to student achievement than the specific concepts addressed. 



Relevance to Policy 

Topical coverage, professional development, and instructional materials and technologies are 
perhaps more amenable to policy setting; the dimension of pedagogy may be less so. Certainly, state 
and local policymakers can establish or recommend topic coverage through curriculum frameworks. 
Professional development criteria, both the amount and type of instructional materials and technology, 
and teacher’s access to and use of such items are also local policy options. These three areas contrast 
with pedagogy, in which decisions have traditionally been the responsibility of the practitioner. Thus, 
if gathering information on these four areas is intended to inform decision making at the federal, 
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state, or district level, attention to topical coverage, professional development, and instructional 
materials may be the most beneficial. 



Appropriateness for the TFS Sample 

NCES surveys differ substantially from other recent work in the characteristics of their 
samples, with important implications for the items included. For the most part, NCES surveys 
(NAEP, SASS, TFS) are directed at a spectrum of teachers in most grade levels and subject areas. 
Indeed, as reported in the March 1994 results from the latest TFS, respondents represented 20 main 
fields of assignment (NCES, March 1994, Table B). In contrast, teacher questionnaires in ongoing 
international (TIMSS) and state-sponsored research (CLASS, SCASS) are directed to the teachers of 
students being examined in certain content areas, with much of that subject-specific inquiry directed 
toward upper-level teachers of math and science. 

A major challenge to creators of survey items is to employ wording with a common meaning 
across all respondents (L. McDonnell, personal communication. May 16, 1994). Wording was the 
single most difficult problem to resolve on the TIMSS project, not only internationally but nationally 
as well, requiring years of negotiation and understanding (Schmidt, June 17, 1994). Although that 
example may be an extreme, commonality of wording is a strong source of concern on any national- 
level questionnaire. For a nationally representative sample, wording problems are certain to escalate 
beyond those experienced in state-level or smaller surveys. This complication may be more true in 
certain educational fields than in others. In content areas and grade levels where pedagogy is rapidly 
changing, the words used to describe that pedagogy is also evolving and in some cases being created. 
As classroom practices and the wording used to describe them become more accepted and common, 
the wording attains greater clarity and universality. For this reason, face validity of questionnaire 
items will be greater for those describing more common elements of the educational process than for 
newly developed, or "reform" practices, and it will be more difficult to include questions about 
"reform" practices on a national survey such as SASS or TFS. 

Questions about instructional materials and technologies, such as all aspects of texts and other 
reading materials, are broadly applicable across disciplines and grade levels with the exception of 
questions focused on specific math- and science-related materials and technologies (especially 
calculator and computer use). Survey items on classroom pedagogy also cover a broad spectrum of 
the teaching/leaming process, with both wording and topics of interest having commonality across 
grade levels and subject matter. Similarly, inquiries into professional development opportunities and 
activities are also broadly applicable across a spectrum of grade levels and subject areas. 
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Many surveys examine topical coverage within courses, but such questions may be unsuitable 
for a broad-based survey such as the TFS that samples across many disciplines and grade levels. 

Such questions have been used primarily in surveys with a targeted audience-subject-specific 
teachers, usually in math or science and often in single grade levels. Limited content overlap across 
subjects and grade levels makes such "content" questions inappropriate for broad-based survey 
samples. 



Level of Specificity 

A final consideration concerns the relative "granularity" of an item— the item’s level of 
specificity. Most questionnaires include some items that seek general information and others that 
probe deeply-each type has benefits and limitations that must be balanced when assembling a set of 
items. For example, items that deeply probe specific aspects of pedagogy can inform educational 
practitioners about detailed elements of practice that could support new understandings about the 
classroom process. Because the number of questions and the respondent burden by necessity must be 
limited, however, indepth questions often preempt items representing a broader range of inquiry, and 
may result in specific and often detailed information about a relatively narrow range of interest. 
Indepth questions (such as course content) are most often directed at a specific target group (such as 
math teachers). The TIMSS math questionnaire, for example, includes 22 items seeking information 
about math content areas covered, plus 18 additional questions at a higher degree of granularity about 
specific concepts covered with topic areas. This level of depth is practical only with a very targeted 
population. Conversely, surveys that seek to gather data on a range of common pedagogical practices 
are less limited in the amount of detail obtainable. Broad surveys have the greatest utility in 
presenting global pictures of the current condition. Of the 31 instruments reviewed, including those 
specifically designed for math teachers, most include a combination of specific and general items. 

The purpose of the survey plays a large role in determining the mix of questions. In general, 
for example, data intended for the purpose of informing public opinion are broad and include 
measures understood by the public. Data used to inform curriculum debate, on the other hand, are 
likely to be more indepth and subject specific. The number of class hours allocated to teaching the 
"conversion of equivalent forms of common fractions" (TIMSS, Teacher Questionnaire, Population 2, 
Question 23), for example, is more relevant to a discussion on curriculum elements than to inf orming 
public opinion about the status of our nation’s schools. To some degree, most surveys seek to inform 
both types of debate and thus include both general and specific items. For such a survey, some 
researchers suggest a balance of items representing issues in depth and in breadth with the proportion 
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of indepth items ranging from one-fourth to one-third of the total (L. Burstein, personal 
communication. May 19, 1994). 

Given the TFS sample that cuts across all grade levels and nearly 20 fields of assignment, 
broad questions from all four interest areas are more appropriate for a diverse group of respondents 
than are narrower questions, although selected elements of pedagogy and professional development 
could be commonly narrow across all respondents. 



Recommendations 

In the previous sections we detailed and evaluated existing efforts to collect information on 
classroom instructional practices. In this section we identify specific areas with potential value for 
national data collection, and suggest methods by which NCES could add to an understanding of the 
classroom process. 

The TFS national probability sample could easily be employed to provide national estimates of 
certain elements of the classroom instructional process. We recommend a broad mix of items 
concentrating on pedagogy while focusing to a lesser extent on professional development and 
materials and technology. We do not recommend including TFS questions on specific course content. 



Pedagogy 



Although the 1991-92 TFS collected no information on pedagogy, the 1990-91 and 1993-94 
SASS-4A and 4B each included questions on the teacher’s level of classroom control (on a scale of 1 
to 6, no control-complete control) over selecting teaching techniques, evaluating and grading students, 
and determining the amount of homework to be assigned. The 1990-91 edition (but not the 1993-94 
version) also had an item on whether homework was assigned during the past week (yes/no). 

Because questions on pedagogy address the heart of the educational process, we recommend 
they form the bulk of the instructional process items on the TFS. We suggest collecting information 
on student actions, teacher actions, instructional organization, student assessment, and homework. 
Items on student actions would query what students actually do in the process of learning; items on 
teacher actions and approaches, such as instructional strategies and practices, questioning techniques, 
and intended learner outcomes, would describe the activities teachers engage in during the process of 
instruction. Information on instructional organization would provide insight on the structure of 
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student and teacher interactions during a typical lesson. Data on student assessment would address 
many current issues on the types and uses of current assessment strategies; information on the 
frequency with which teachers assign homework, its type and length, and the uses to which teachers 
put the completed assignments would provide insight on an issue of popular interest. 

Selected from the VNCI, RAND/UCLA, CLASS, SCASS, and RUC questionnaires, these 
items could be combined for simplicity, refined for consistency, and in some cases, leveraged to 
provide additional related information. 

For the maximum number of items, if not all, we recommend using frequency of use response 
options rather than yes/no response options because these options provide more information at 
minimal added instrument space or respondent time. Furthermore, the frequency of use should be 
defined in clusters of time, such as "daily/1-2 each week/1-2 each month" or "0-1 periods/2-5 
periods/5-10 periods." 



Professional Development 

Analyzing professional development via information from the 1991-92 TFS is limited to 
information from those teachers currently enrolled in a degree program; the 1990-91 SASS-4A and 
4B collected information on attendance at workshops or inservice training. The 1993-94 SASS 
expands on that approach with questions about support for development activities, attendance at 
programs with a specific focus (such as educational technology, content methods, student assessment, 
cooperative learning), and its impact on their teaching. 

We recommend adding a question covering teacher participation in less formal development 
activities, such as time available for reading professional journals, lesson planning with colleagues, 
and long-range curriculum development planning with other teachers. Such an item could be adapted 
from elements of the RUC questionnaire. 



Materials and Technology 

For the most part, the materials and technologies of teaching are either grade-level or subject- 
area specific and difficult to phrase generically for teachers of all fields. Perhaps for this same 
reason, the current information obtained on materials and technologies from both TFS and SASS 
questionnaires is minimal. Information from the most recent TFS about the level of teaching 
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materials and technology is obtained from responses (in terms of 4 levels of satisfaction) to one of 22 
aspects of teaching: "availability of resources and materials/equipment for your classroom." The 
1993-94 SASS-4A included two questions on materials: one about teacher control over selecting texts 
and other materials (6-point scale); the other about the teacher’s perception whether the necessary 
materials are provided for the staff (4-point scale). Information on library and media center materials 
and technology is collected in the SASS family of six library questionnaires. 

We recommend collecting information on the availability and uses of specific materials such 
as texts, workbooks and ditto sheets, and other supplementary printed materials. Items framed to cut 
across disciplines would provide additional interesting data on the frequency and purpose of calculator 
and computer usage and fill a gap in both SASS and TFS instruments. If possible, such questions 
might profitably be incorporated into pedagogy questions on student actions. TIMSS and RUC 
questionnaires are sources for questions on texts and other printed materials; SRA, RUC, and TIMSS 
provide options for items on calculator/computer use. 



Topic Coverage within Courses 

Although most questionnaires sought information about the specific topics taught within 
designated courses, the topic’s emphasis within that course, and the allocation of classroom time 
devoted to each topic, such questionnaires are directed to teachers of a specific subject matter (such as 
math) at a specific grade level (such as eighth grade). Because the TFS is a nationally representative 
sample encompassing teachers of more than 20 fields of assignment, it should not attempt to collect 
information about specific content topics or concepts within topics, because the depth of questions 
necessary to obtain meaningful information on course coverage is inappropriate to the sample. Many 
of the skills and processes included above in pedagogy, however, are closely related to content and 
could be considered distinct curricular elements providing information to those interested in specific 
topics within courses. Writing reports, and creating, using, and explaining charts, graphs, or tables 
are examples of distinct student skills with application in many content areas. Information about the 
frequency with which teachers teach these skills and require their application would be useful content- 
related information. 



Conclusion 

The extent, appropriateness, and accessibility of learning opportunities for students has 
become a central issue in the national discourse on how to define, measure, and improve the quality 



of education throughout the nation. Nationally representative data that paint an accurate picture of 
students in the classroom would improve the basis on which the current status of education could be 
measured and better inform future federal policy. As a source of nationally representative 
information from the teacher corps, the Teacher Followup Survey has great potential for amassing 
such information from practicing teachers. Information could be collected in the critical dimensions 
of pedagogy, professional development, and materials and technology, specifically on student and 
teacher actions and interactions within the classroom, uses of homework, methods of student 
assessment, and the tools of the teaching trade. Incorporating such items in the 1994-95 Teacher 
Followup Survey could provide insightful information to federal, state, and local policymakers about 
issues of growing national concern. 



Appendix A 

Listing of Survey Instruments Reviewed 



National Center for Education Statistics (NCES): Three NCES data collection strategies 
provide information on the condition and processes of elementary and secondary schooling: the 
National Assessment of Educational Progress (NAEP), the National Education Longitudinal Study of 
1988 (NELS.88), the Schools and Staffing Survey (SASS), and the SASS Teacher Followup Survey 
(TFS). 



NAEP, Gr 4 Reading, U.S. History, World Geography Teacher Questionnaire, 93-94 
NAEP, Gr 8 Reading Teacher Questionnaire, 93-94 

NELS, Teacher Questionnaire NELS:88 Base year 

NELS, First Followup, Teacher Questionnaire, Math version (1990) 

NELS, First Followup, Teacher Questionnaire, Science version (1990) 

NELS, First Followup, Teacher Questionnaire, History version (1990) 

NELS, First Followup, Teacher Questionnaire, English version (1990) 

NELS, Second Followup, Teacher Questionnaire, (1992) 

S ASS-4 A, Public School Teacher Questionnaire, 1993-94 school year 
SASS-4A, Public School Teacher Questionnaire, 1990-91 school year 
SASS-4B, Private School Teacher Questionnaire, 1990-91 school year 

LS-1A, Public School Library Media Center Questionnaire, 1993-94 school year 
LS-1B, Private School Library Media Center Questionnaire, 1993-94 school year 
LS-1C, Indian School Library Media Center Questionnaire, 1993-94 school year 
LS-2A, Public School Library Media Specialist/Librarian Questionnaire, 

1993-94 school year 

LS-2B, Private School Library Media Specialist/Librarian Questionnaire, 

1993-94 school year 

LS-2C, Indian School Library Media Specialist/Librarian Questionnaire, 

1993-94 school year 

TFS-2, Former Teacher Questionnaire, 1991-92 school year 
TFS-3, Current Teacher Questionnaire, 1991-92 school year 
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RAND/UCLA: Three projects form the basis of this team’s work. The School Reform 
Assessment Project (SRA) was a design study to refine the quality of coursework indicators and to 
measure the effects of major curriculum policies. Data were collected during the 1988-89 school year 
from seven high schools in Georgia and California. The purpose of Validating National Curriculum 
Indicators (VNCI) was to collect data about what is actually taught in classrooms with which to 
validate data collected via routine surveys. Data were collected during the 1992-93 school year from 
88 high school science and math teachers. Items developed for the California Learning Assessment 
System (CLAS) gather information from both students and teachers on the opportunities provided 
students to learn the math addressed in the CLAS math performance assessment. Data were collected 
from students in grades 4, 8, and 10 during the 1992-93 school year. 

SRA, Teacher Questionnaire, Part I 
SRA, Teacher Questionnaire, Part II 

VNCI, Initial Teacher Survey, RAND/UCLA 
VNCI, Daily Log 

VNCI, Mathematics Teacher Survey Form 1: Math I 
VNCI, Mathematics Teacher Survey Form 2: Math II 

CLAS, Session Header, 1994, Grades 4, 8, 10 
CLAS, Class Header, 1993, Grades 4, 8, 10 



Reform Up Close (RUC): RUC is a study of high school mathematics and science in six 
states, 12 districts, and 18 schools seeking to document instructional practices and classroom 
opportunities to learn the curriculum. Data were collected during the 1990-91 school year. 

Daily Log, dated 8/2/90 
Weekly Questionnaire, dated 1 1/2/89 
Mathematics/Science Teacher Questionnaire 
Course-Specific Mathematics Teacher Questionnaire, Part II 



State Collaborative on Assessment of Student Standards (SCASS): Project directed by the 
Council of Chief State School Officers to develop measures of students’ opportunity to learn science 
for use by states, school districts, and classroom teachers in program improvement. The project is in 
its preliminary phase. 

Teacher Survey, RAND, Spring 1994 

Opportunity to Learn Teacher Questionnaire, (Feb 23, 1994) 

Teacher Questionnaire, SCASS Science Field Trial (Spring 1994) 

Models for Collecting and Using Data on Opportunity to Learn at the State Level: 
OTL Options for the CCSSO SCASS Science Assessment (June 13, 1994) 
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Third International Mathematics and Science Study /Study of Mathematics and Science 
Opportunity (TIMSS/SMSO): An integrated study of the content, performance expectations, and 
perspectives of math and science teachers. The international study incorporates three populations: 9- 
and 13-year-olds, and students in their final year of secondary schooling. SMSO is a companion 
study of the extent to which an opportunity was provided in the classroom to study the material 
needed to answer the questions on the TIMSS. Data will be collected during fall 1994 in the southern 
hemisphere and spring 1995 in the northern hemisphere. 

School Questionnaire, Population 1, Review Version, May 1994 
Teacher Questionnaire, Population 2, Mathematics, Review Version, May 1994 
Teacher Questionnaire, Population 2, Science, Review Version, May 1994 
Appendices, p. 37-129 
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